Attribute shaded Maven classes to their dependency packages by chinyeungli · Pull Request #2249 · aboutcode-org/scancode.io

chinyeungli · 2026-10-02T10:43:06Z

Issues

Closes: maven-heaven: Correlate relocated classes in shaded JARs to their originating packages #2247

Changes

Maven Shade relocates dependency classes under a new package prefix, so they have no matching source in the development codebase. These classes were previously flagged as missing sources, which is misleading because they come from a dependency.

This PR parses the main POM for maven-shade-plugin entries and reverses them to recover the original class names. Then, it matches each unmatched .class against the Maven dependency packages using the groupId, artifactId, and the class's package path (with a scoring system). If the match meets the required score, it will then assign the matched resource's status as "shaded-class".

Notes

Classes that cannot be matched unambiguously are intentionally left as requires-review rather than guessed.

Checklist

I have read the contributing guidelines
I have linked an existing issue above
I have added unit tests covering the new code
I have reviewed and understood every line of this PR

tdruez

@chinyeungli The relocation parsing makes sense, but the scoring runs on every unmapped .class of the to/ side, not only on the shaded ones. Unmapped classes of the main project get attributed to any dependency from the same group. For example, org.apache.hadoop.mapreduce.Job matches pkg:maven/org.apache.hadoop/hadoop-common with a score of 5 and is flagged as shaded-class, which is worse than leaving it as requires-review.

What about limiting the matching to the classes under a shadedPattern when relocations are available, and excluding the main package namespace otherwise?
In which cases are the dependency DiscoveredPackage present in the project? Dependencies declared in the POM are created as DiscoveredDependency. The tests create those packages directly, so I'm not sure this path runs on a real shaded JAR.
How were the weights and MIN_MATCH_SCORE = 5 chosen? Were they tested on shaded JARs other than htrace-core?
The new step is never executed by the test suite: test_scanpipe_scan_maven_package_single_file runs without D2D, and the step only runs with D2D enabled. We still need a pipeline test with D2D, ideally on a real shaded JAR, so the relocation reversal is covered too (test_scanpipe_maven_map_shaded_classes_attributions creates no main POM).
Missing tests for: the POM parsing failure in get_maven_shade_relocations, the "no POM found" case in get_main_maven_pom, and the early return without dependency packages in map_shaded_classes_to_maven_packages.

See my inline comments for the rest.

chinyeungli · 2026-10-07T06:08:29Z

What about limiting the matching to the classes under a shadedPattern when relocations are available, and excluding the main package namespace otherwise?

It is now handled in _get_original_fqn() that check if relocations are available and only work on classes under shadedPattern.

In which cases are the dependency DiscoveredPackage present in the project? Dependencies declared in the POM are created as DiscoveredDependency. The tests create those packages directly, so I'm not sure this path runs on a real shaded JAR.

Right. I updated the code and tests.

How were the weights and MIN_MATCH_SCORE = 5 chosen? Were they tested on shaded JARs other than htrace-core?

It's chosen by the following minimum scenario:

a second matching artifact segment (1 group + 2 artifact = 1 + 4 = 5)
two extra matching group segments (3 group + 1 artifact = 3 + 2 = 5)
a fuzzy match (1 group + 1 artifact + fuzzy = 1 + 2 + 3 = 6)

I also checked pkg:maven/org.apache.activemq/artemis-commons@2.31.1 and the score 5 seems to work.
Note that this package not only using shadedPattern, but also artifactSet for shaded files

<artifactSet>
   <includes>
      <include>org.apache.johnzon:johnzon-core</include>
      <include>jakarta.json:jakarta.json-api</include>
   </includes>
</artifactSet>

https://repo1.maven.org/maven2/org/apache/activemq/artemis-commons/2.31.1/artemis-commons-2.31.1.pom

Created #2251 for follow up.

The new step is never executed by the test suite: test_scanpipe_scan_maven_package_single_file runs without D2D, and the step only runs with D2D enabled. We still need a pipeline test with D2D, ideally on a real shaded JAR, so the relocation reversal is covered too (test_scanpipe_maven_map_shaded_classes_attributions creates no main POM).

Added test_scanpipe_scan_maven_package_d2d_shaded_classes, and it uses the real shaded-pom.xml

Missing tests for: the POM parsing failure in get_maven_shade_relocations, the "no POM found" case in get_main_maven_pom, and the early return without dependency packages in map_shaded_classes_to_maven_packages.

I think I have these covered.

tdruez

@chinyeungli Most of the previous points are addressed. The remaining concern is still wrong attributions, see my comments on the scoring, the version deduplication and the no-relocation path.

The new pipeline test depends on internet access: it downloads htrace-core from Maven Central and runs the full D2D. I'd mock fetch_inputs and use a small shaded JAR fixture.
When the match is an unresolved DiscoveredDependency, the class gets the shaded-class status but is not assigned to any package. The attribution only lives in extra_data, so it won't show in the package resources or in the SBOM outputs. Is this the intended result? Should a DiscoveredPackage be created for the dependency in that case?
Missing tests: get_maven_shade_relocations without a main POM (pom_resource is None), and match_shaded_class_to_package with a candidate that has both signals but a score below MIN_MATCH_SCORE.

* Added code to handle shade relocated dependency classes * Match each unmatched .class against the Maven dependency packages using the groupId, artifactId, and the class's package path. Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Better code structures, tests, docstrings etc. Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Use mock data for pipeline test * Created DiscoveredPackage for unresolved DiscoveredDependency * Add tests * Increase MIN_MATCH_SCORE to 6 * etc... Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez

@chinyeungli Most of the previous comments are addressed. The package creation for unresolved dependencies has a bug, see my inline comments.

test_scanpipe_maven_get_maven_dependency_packages_dedupes_by_name was removed, so the deduplication of a DiscoveredPackage and an unresolved DiscoveredDependency for the same artifact is not tested anymore. This is the real shaded JAR case: the package comes from META-INF/maven/ and the dependency from the main POM.

tdruez · 2026-10-21T06:19:39Z

+    Return "candidates" deduplicated by "namespace" and "name", excluding the
+    main package.


Docstring out of sync: the version is part of the key now.

Suggested change

Return "candidates" deduplicated by "namespace" and "name", excluding the

main package.

Return "candidates" deduplicated by "namespace", "name" and "version",

excluding the main package.

tdruez · 2026-10-21T06:19:39Z

+            return original_pattern + fqn[len(shaded_pattern) :]
+
+    return None


return None is redundant and not consistent with the codebase style.

Suggested change

return original_pattern + fqn[len(shaded_pattern) :]

return None

return original_pattern + fqn[len(shaded_pattern) :]

tdruez · 2026-10-21T06:19:39Z

+# Reaching 6 ensures a highly reliable match by requiring one of:
+#   - multiple matching artifact and group segments (2 group + 2 artifact = 2 + 4 = 6)
+#   - a fuzzy match (1 group + 1 artifact + fuzzy = 1 + 2 + 3 = 6)


Raising the threshold moves the problem by one segment: 4 group + 1 artifact also reaches 6, which this comment doesn't list. When the groupId is also the Java package, any class under it matches any artifact of the group. With pkg:maven/com.fasterxml.jackson.core/jackson-databind:

>>> score_dependency("com.fasterxml.jackson.core", jackson_databind) (6, {'group', 'artifact'})

jackson-core is a transitive dependency, so with only jackson-databind declared, the shaded jackson-core classes go to jackson-databind. What about requiring a match, exact or fuzzy, on the last artifact segment (databind, logging, annotations), which is the specific part? To be tested on more JARs though.

tdruez · 2026-10-21T06:19:39Z

+def _get_original_fqn(fqn, relocations):
+    """
+    Return the original fully qualified class name for a shaded class, or
+    None when the class should not be considered for matching.
+    """
+    if relocations:
+        return apply_shade_relocations(fqn, relocations)


Not needed anymore: apply_shade_relocations already returns None when relocations is empty. It's also called with a Java package now, so the name is misleading. I'd drop it and call apply_shade_relocations directly, see below.

tdruez · 2026-10-21T06:19:39Z

+    otherwise.
+    """
+    fqn = get_java_fqn_from_class_path(resource.path)
+    java_package = fqn.rsplit(".", 1)[0] if "." in fqn else ""


rpartition returns an empty string when there is no dot:

Suggested change

java_package = fqn.rsplit(".", 1)[0] if "." in fqn else ""

java_package = fqn.rpartition(".")[0]

tdruez · 2026-10-21T06:19:39Z

+    if isinstance(matched_package, DiscoveredDependency):
+        package_data = {
+            "type": matched_package.type,
+            "namespace": matched_package.namespace,
+            "name": matched_package.name,
+            "version": matched_package.version,
+        }
+        created_package = DiscoveredPackage.create_from_data(
+            resource.project, package_data
+        )
+        matched_package.resolved_to_package = created_package
+        matched_package.save()
+        matched_package = created_package


The cache holds the DiscoveredDependency, so a new DiscoveredPackage is created for every class matched to it, not once per dependency: N shaded classes give N duplicated packages, and resolved_to_package is overwritten each time. I'd create the package once, in a dedicated function:

def get_or_create_dependency_package(dependency): """Return the resolved package of the "dependency", created when missing.""" if not dependency.resolved_to_package: package_data = { "type": dependency.type, "namespace": dependency.namespace, "name": dependency.name, "version": dependency.version, } package = DiscoveredPackage.create_from_data(dependency.project, package_data) dependency.update(resolved_to_package=package) return dependency.resolved_to_package

This also replaces the bare save() with update(). A test with 2 classes matched to the same unresolved dependency would catch it.

tdruez · 2026-10-21T06:19:39Z

+    def setUp(self):
+        self.project = Project.objects.create(name="Analysis")


Same as my previous comment: the existing tests are still modified (lines 172 and 190). With make_project() from scanpipe.tests here, they can stay as they are on main.

Suggested change

def setUp(self):

self.project = Project.objects.create(name="Analysis")

def setUp(self):

self.project = make_project()

tdruez · 2026-10-21T06:19:40Z

+        make_package(
+            project,
+            package_url="pkg:maven/org.apache.htrace/htrace-core@4.0.0-incubating",
+            type="maven",
+            namespace="org.apache.htrace",
+            name="htrace-core",
+            version="4.0.0-incubating",
+        )


Same as my previous comment: make_package sets type, namespace, name and version from package_url, so those 4 arguments are not needed, here and in the other make_package calls. The pipeline test already uses the short form.

Suggested change

make_package(

project,

package_url="pkg:maven/org.apache.htrace/htrace-core@4.0.0-incubating",

type="maven",

namespace="org.apache.htrace",

name="htrace-core",

version="4.0.0-incubating",

)

make_package(

project,

package_url="pkg:maven/org.apache.htrace/htrace-core@4.0.0-incubating",

)

tdruez · 2026-10-21T06:19:40Z

+        # Only run the steps needed to extract resources, run D2D, and map
+        # shaded classes. Skipping run_scan / fetch_and_scan_remote_pom /
+        # load_inventory_from_toolkit_scan / make_summary_from_scan_results
+        # avoids both network calls and the toolkit scan.
+        run = project1.add_pipeline(pipeline_name)
+        run.update(
+            selected_steps=[
+                "check_input_and_return_purl",
+                "fetch_inputs",
+                "d2d_check",
+                "collect_input_info",
+                "extract_input",
+                "extract_archives",
+                "maven_d2d_steps",
+                "map_shaded_classes_to_maven_packages",
+                "update_package_license_from_resource_if_missing",
+            ],
+        )
+        run.refresh_from_db()


Why skip run_scan and load_inventory_from_toolkit_scan? fetch_and_scan_remote_pom doesn't fetch anything when a pom.xml is in the codebase, so the full pipeline should run offline with this JAR. The POM dependencies would then be created as DiscoveredDependency, as on a real shaded JAR, which also covers the package creation for unresolved dependencies. Neither the remove_packages_without_resources mock nor the make_package calls would be needed. To be tested though.

tdruez · 2026-10-21T06:19:40Z

+        tmpdir = tempfile.mkdtemp(prefix="scanpipe-mvn-")
+        self.addCleanup(shutil.rmtree, tmpdir, True)
+
+        src_path = os.path.join(tmpdir, "htrace-core-4.0.0-incubating-sources.jar")
+        bin_path = os.path.join(tmpdir, "htrace-core-4.0.0-incubating.jar")
+
+        with zipfile.ZipFile(src_path, "w") as zf:
+            zf.writestr("Dummy.java", b"public class Dummy {}")
+
+        with zipfile.ZipFile(bin_path, "w") as zf:


pathlib over os.path, and a descriptive name over zf, for example jar_file.

chinyeungli requested a review from tdruez October 2, 2026 10:43

chinyeungli mentioned this pull request Oct 5, 2026

maven-heaven: Correlate relocated classes in shaded JARs to their originating packages #2247

Open

tdruez requested changes Oct 5, 2026

View reviewed changes

chinyeungli requested a review from tdruez October 7, 2026 08:04

tdruez requested changes Oct 15, 2026

View reviewed changes

chinyeungli requested a review from tdruez October 21, 2026 05:03

chinyeungli added 4 commits October 21, 2026 14:22

Address PR feedback #2247

23b0dbf

* Better code structures, tests, docstrings etc. Signed-off-by: Chin Yeung Li <tli@nexb.com>

Address PR feedback #2247

0e03c34

* Use mock data for pipeline test * Created DiscoveredPackage for unresolved DiscoveredDependency * Add tests * Increase MIN_MATCH_SCORE to 6 * etc... Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update code formatting #2247

923973e

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli force-pushed the 2247_corelated_shaded_files branch from b555d2c to 923973e Compare October 21, 2026 06:22

tdruez requested changes Oct 21, 2026

View reviewed changes

		Return "candidates" deduplicated by "namespace" and "name", excluding the
		main package.

		return original_pattern + fqn[len(shaded_pattern) :]

		return None

	java_package = fqn.rsplit(".", 1)[0] if "." in fqn else ""
	java_package = fqn.rpartition(".")[0]

		def setUp(self):
		self.project = Project.objects.create(name="Analysis")

Conversation

chinyeungli commented Oct 2, 2026

Issues

Changes

Notes

Checklist

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chinyeungli commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chinyeungli commented Oct 7, 2026 •

edited

Loading